UME-21 local evaluation initiatives: contributions and challenges.

BACKGROUND AND OBJECTIVES The 18 medical schools involved in the UME-21 initiative developed innovative curricula and evaluation strategies. While there was significant variation in how schools approached the evaluation process, there were common methodological issues and challenges affecting the reliability of scores and validity of interpretations regarding outcomes. This paper explores these issues and challenges, using experiences from selected UME-21 schools. METHODS Four evaluation issues and strategies are discussed: instrument development, study design, process evaluation using formative evaluation methods, and qualitative strategies. Within each discussion, examples from a UME-21 school are presented. RESULTS The four evaluation strategies offered the flexibility to match local evaluation needs with an effective approach to evaluations. CONCLUSIONS/IMPLICATIONS The school-level evaluation requirements by the UME-21 initiative provided schools the flexibility to design individualized evaluation strategies, yet also encouraged collaboration among evaluators. While this strategy resulted in many successes at the school level, it also served to identify common methodological challenges that can be used as a guide for other schools in implementing and evaluating curricula.